{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Install and Connect\n",
    "\n",
    "To get started, we'll need to connect to our Elastic deployment using the Python client.\n",
    "Because we're using an Elastic Cloud deployment, we'll use the **Cloud ID** to identify our deployment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -qU elasticsearch requests openai"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need to import the modules we need. 🔐 NOTE: getpass enables us to securely prompt the user for credentials without echoing them to the terminal, or storing it in memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from elasticsearch import Elasticsearch, helpers, exceptions\n",
    "from elasticsearch.helpers import BulkIndexError\n",
    "from getpass import getpass\n",
    "import time\n",
    "import json as JSON\n",
    "from openai import AzureOpenAI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can instantiate the Python Elasticsearch client.\n",
    "\n",
    "First we prompt the user for their password and Cloud ID. Then we create a client object that instantiates an instance of the Elasticsearch class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://www.elastic.co/search-labs/tutorials/install-elasticsearch/elastic-cloud#finding-your-cloud-id\n",
    "ELASTIC_CLOUD_ID = getpass(\"Elastic Cloud ID: \")\n",
    "\n",
    "# https://www.elastic.co/search-labs/tutorials/install-elasticsearch/elastic-cloud#creating-an-api-key\n",
    "ELASTIC_API_KEY = getpass(\"Elastic Api Key: \")\n",
    "\n",
    "# Create the client instance\n",
    "client = Elasticsearch(\n",
    "    # For local development\n",
    "    # hosts=[\"http://localhost:9200\"]\n",
    "    cloud_id=ELASTIC_CLOUD_ID,\n",
    "    api_key=ELASTIC_API_KEY,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Enable Telemetry\n",
    "\n",
    "Knowing that you are using this notebook helps us decide where to invest our efforts to improve our products. We would like to ask you that you run the following code to let us gather anonymous usage statistics. See [telemetry.py](https://github.com/elastic/elasticsearch-labs/blob/main/telemetry/telemetry.py) for details. Thank you!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Telemetry enabled for \"09-geospatial-search\". Thank you!\n"
     ]
    }
   ],
   "source": [
    "!curl -O -s https://raw.githubusercontent.com/elastic/elasticsearch-labs/main/telemetry/telemetry.py\n",
    "from telemetry import enable_telemetry\n",
    "\n",
    "client = enable_telemetry(client, \"09-geospatial-search\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test the Client\n",
    "Before you continue, confirm that the client has connected with this test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'instance-0000000001', 'cluster_name': 'c2826e6e8ae74e46b5911a3c89af31cf', 'cluster_uuid': 'jpiDThQURAKY93YeiDkfdA', 'version': {'number': '8.13.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da95df118650b55a500dcc181889ac35c6d8da7c', 'build_date': '2024-05-06T22:04:45.107454559Z', 'build_snapshot': False, 'lucene_version': '9.10.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}\n"
     ]
    }
   ],
   "source": [
    "print(client.info())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Refer to https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/connecting.html#connect-self-managed-new to learn how to connect to a self-managed deployment.\n",
    "\n",
    "Read https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/connecting.html#connect-self-managed-new to learn how to connect using API keys."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Download and Deploy ELSER Model\n",
    "\n",
    "In this example, we are going to download and deploy the ELSER model in our ML node. Make sure you have an ML node in order to run the ELSER model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model deleted successfully, We will proceed with creating one\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "ObjectApiResponse({'model_id': '.elser_model_2_linux-x86_64', 'model_type': 'pytorch', 'model_package': {'packaged_model_id': 'elser_model_2_linux-x86_64', 'model_repository': 'https://ml-models.elastic.co', 'minimum_version': '11.0.0', 'size': 274756282, 'sha256': 'be69211494bf9cdc57a7aa0ee06814fcccf999407237816c9d9f0963858e2a61', 'metadata': {}, 'tags': [], 'vocabulary_file': 'elser_model_2_linux-x86_64.vocab.json', 'platform_architecture': 'linux-x86_64'}, 'platform_architecture': 'linux-x86_64', 'created_by': 'api_user', 'version': '12.0.0', 'create_time': 1715935799552, 'model_size_bytes': 0, 'estimated_operations': 0, 'license_level': 'platinum', 'description': 'Elastic Learned Sparse EncodeR v2 optimized for linux-x86_64', 'tags': ['elastic'], 'metadata': {}, 'input': {'field_names': ['text_field']}, 'inference_config': {'text_expansion': {'vocabulary': {'index': '.ml-inference-native-000002'}, 'tokenization': {'bert': {'do_lower_case': True, 'with_special_tokens': True, 'max_sequence_length': 512, 'truncate': 'first', 'span': -1}}}}, 'location': {'index': {'name': '.ml-inference-native-000002'}}})"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# delete model if already downloaded and deployed\n",
    "try:\n",
    "    client.ml.delete_trained_model(model_id=\".elser_model_2_linux-x86_64\", force=True)\n",
    "    print(\"Model deleted successfully, We will proceed with creating one\")\n",
    "except exceptions.NotFoundError:\n",
    "    print(\"Model doesn't exist, but We will proceed with creating one\")\n",
    "\n",
    "# Creates the ELSER model configuration. Automatically downloads the model if it doesn't exist.\n",
    "client.ml.put_trained_model(\n",
    "    model_id=\".elser_model_2_linux-x86_64\", input={\"field_names\": [\"text_field\"]}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above command will download the ELSER model. This will take a few minutes to complete. Use the following command to check the status of the model download."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ELSER Model is downloaded but not ready to be deployed.\n",
      "ELSER Model is downloaded but not ready to be deployed.\n",
      "ELSER Model is downloaded but not ready to be deployed.\n",
      "ELSER Model is downloaded but not ready to be deployed.\n",
      "ELSER Model is downloaded but not ready to be deployed.\n",
      "ELSER Model is downloaded and ready to be deployed.\n"
     ]
    }
   ],
   "source": [
    "while True:\n",
    "    status = client.ml.get_trained_models(\n",
    "        model_id=\".elser_model_2_linux-x86_64\", include=\"definition_status\"\n",
    "    )\n",
    "\n",
    "    if status[\"trained_model_configs\"][0][\"fully_defined\"]:\n",
    "        print(\"ELSER Model is downloaded and ready to be deployed.\")\n",
    "        break\n",
    "    else:\n",
    "        print(\"ELSER Model is downloaded but not ready to be deployed.\")\n",
    "    time.sleep(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the model is downloaded, we can deploy the model in our ML node. Use the following command to deploy the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ELSER Model is currently being deployed.\n",
      "ELSER Model is currently being deployed.\n",
      "ELSER Model has been successfully deployed.\n"
     ]
    }
   ],
   "source": [
    "# Start trained model deployment if not already deployed\n",
    "client.ml.start_trained_model_deployment(\n",
    "    model_id=\".elser_model_2_linux-x86_64\", number_of_allocations=1, wait_for=\"starting\"\n",
    ")\n",
    "\n",
    "while True:\n",
    "    status = client.ml.get_trained_models_stats(\n",
    "        model_id=\".elser_model_2_linux-x86_64\",\n",
    "    )\n",
    "    if status[\"trained_model_stats\"][0][\"deployment_stats\"][\"state\"] == \"started\":\n",
    "        print(\"ELSER Model has been successfully deployed.\")\n",
    "        break\n",
    "    else:\n",
    "        print(\"ELSER Model is currently being deployed.\")\n",
    "    time.sleep(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing Documents with ELSER\n",
    "In order to use ELSER on our Elastic Cloud deployment we'll need to create an ingest pipeline that contains an inference processor that runs the ELSER model. Let's add that pipeline using the [put_pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/master/put-pipeline-api.html) method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ObjectApiResponse({'acknowledged': True})"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.ingest.put_pipeline(\n",
    "    id=\"elser-ingest-pipeline\",\n",
    "    description=\"Ingest pipeline for ELSER\",\n",
    "    processors=[\n",
    "        {\"html_strip\": {\"field\": \"name\", \"ignore_failure\": True}},\n",
    "        {\"html_strip\": {\"field\": \"description\", \"ignore_failure\": True}},\n",
    "        {\"html_strip\": {\"field\": \"amenities\", \"ignore_failure\": True}},\n",
    "        {\"html_strip\": {\"field\": \"host_about\", \"ignore_failure\": True}},\n",
    "        {\n",
    "            \"inference\": {\n",
    "                \"model_id\": \".elser_model_2_linux-x86_64\",\n",
    "                \"input_output\": [\n",
    "                    {\"input_field\": \"name\", \"output_field\": \"name_embedding\"}\n",
    "                ],\n",
    "                \"ignore_failure\": True,\n",
    "            }\n",
    "        },\n",
    "        {\n",
    "            \"inference\": {\n",
    "                \"model_id\": \".elser_model_2_linux-x86_64\",\n",
    "                \"input_output\": [\n",
    "                    {\n",
    "                        \"input_field\": \"description\",\n",
    "                        \"output_field\": \"description_embedding\",\n",
    "                    }\n",
    "                ],\n",
    "                \"ignore_failure\": True,\n",
    "            }\n",
    "        },\n",
    "        {\n",
    "            \"inference\": {\n",
    "                \"model_id\": \".elser_model_2_linux-x86_64\",\n",
    "                \"input_output\": [\n",
    "                    {\"input_field\": \"amenities\", \"output_field\": \"amenities_embedding\"}\n",
    "                ],\n",
    "                \"ignore_failure\": True,\n",
    "            }\n",
    "        },\n",
    "        {\n",
    "            \"inference\": {\n",
    "                \"model_id\": \".elser_model_2_linux-x86_64\",\n",
    "                \"input_output\": [\n",
    "                    {\n",
    "                        \"input_field\": \"host_about\",\n",
    "                        \"output_field\": \"host_about_embedding\",\n",
    "                    }\n",
    "                ],\n",
    "                \"ignore_failure\": True,\n",
    "            }\n",
    "        },\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing the AirBnB Listings\n",
    "\n",
    "Next up we need to prepare the index. We will map everything as keyword unless otherwise specified. We will also map the `name` and the `description` of the listing as `sparse_vectors` using ELSER."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ObjectApiResponse({'acknowledged': True, 'shards_acknowledged': True, 'index': 'airbnb-listings'})"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.indices.delete(index=\"airbnb-listings\", ignore_unavailable=True)\n",
    "client.indices.create(\n",
    "    index=\"airbnb-listings\",\n",
    "    settings={\"index\": {\"default_pipeline\": \"elser-ingest-pipeline\"}},\n",
    "    mappings={\n",
    "        \"dynamic_templates\": [\n",
    "            {\n",
    "                \"stringsaskeywords\": {\n",
    "                    \"match\": \"*\",\n",
    "                    \"match_mapping_type\": \"string\",\n",
    "                    \"mapping\": {\"type\": \"keyword\"},\n",
    "                }\n",
    "            }\n",
    "        ],\n",
    "        \"properties\": {\n",
    "            \"host_about_embedding\": {\"type\": \"sparse_vector\"},\n",
    "            \"amenities_embedding\": {\"type\": \"sparse_vector\"},\n",
    "            \"description_embedding\": {\"type\": \"sparse_vector\"},\n",
    "            \"name_embedding\": {\"type\": \"sparse_vector\"},\n",
    "            \"location\": {\"type\": \"geo_point\"},\n",
    "        },\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Downloading the AirBnB data\n",
    "\n",
    "next up we will download the AirBnB listings csv and upload it to Elasticsearch. This can take a couple of minutes! The AirBnB listing is roughly ~80mb of CSV expanded and roughly 40.000 documents. In the code below we added an if condition to only process the first 5.000 documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing document 0\n",
      "Processing document 250\n",
      "Processing document 500\n",
      "Processing document 750\n",
      "Processing document 1000\n",
      "Processing document 1250\n",
      "Processing document 1500\n",
      "Processing document 1750\n",
      "Processing document 2000\n",
      "Processing document 2250\n",
      "Processing document 2500\n",
      "Processing document 2750\n",
      "Processing document 3000\n",
      "Processing document 3250\n",
      "Processing document 3500\n",
      "Processing document 3750\n",
      "Processing document 4000\n",
      "Processing document 4250\n",
      "Processing document 4500\n",
      "Processing document 4750\n",
      "Processing document 5000\n",
      "Processing document 5250\n",
      "Processing document 5500\n",
      "Processing document 5750\n",
      "Processing document 6000\n",
      "Processing document 6250\n",
      "Processing document 6500\n",
      "Processing document 6750\n",
      "Processing document 7000\n",
      "Processing document 7250\n",
      "Processing document 7500\n",
      "Processing document 7750\n",
      "Processing document 8000\n",
      "Processing document 8250\n",
      "Processing document 8500\n",
      "Processing document 8750\n",
      "Processing document 9000\n",
      "Processing document 9250\n",
      "Processing document 9500\n",
      "Processing document 9750\n",
      "Processing document 10000\n",
      "Processing document 10250\n",
      "Processing document 10500\n",
      "Processing document 10750\n",
      "Processing document 11000\n",
      "Processing document 11250\n",
      "Processing document 11500\n",
      "Processing document 11750\n",
      "Processing document 12000\n",
      "Processing document 12250\n",
      "Processing document 12500\n",
      "Processing document 12750\n",
      "Processing document 13000\n",
      "Processing document 13250\n",
      "Processing document 13500\n",
      "Processing document 13750\n",
      "Processing document 14000\n",
      "Processing document 14250\n",
      "Processing document 14500\n",
      "Processing document 14750\n",
      "Processing document 15000\n",
      "Processing document 15250\n",
      "Processing document 15500\n",
      "Processing document 15750\n",
      "Processing document 16000\n",
      "Processing document 16250\n",
      "Processing document 16500\n",
      "Processing document 16750\n",
      "Processing document 17000\n",
      "Processing document 17250\n",
      "Processing document 17500\n",
      "Processing document 17750\n",
      "Processing document 18000\n",
      "Processing document 18250\n",
      "Processing document 18500\n",
      "Processing document 18750\n",
      "Processing document 19000\n",
      "Processing document 19250\n",
      "Processing document 19500\n",
      "Processing document 19750\n",
      "Processing document 20000\n",
      "Processing document 20250\n",
      "Processing document 20500\n",
      "Processing document 20750\n",
      "Processing document 21000\n",
      "Processing document 21250\n",
      "Processing document 21500\n",
      "Processing document 21750\n",
      "Processing document 22000\n",
      "Processing document 22250\n",
      "Processing document 22500\n",
      "Processing document 22750\n",
      "Processing document 23000\n",
      "Processing document 23250\n",
      "Processing document 23500\n",
      "Processing document 23750\n",
      "Processing document 24000\n",
      "Processing document 24250\n",
      "Processing document 24500\n",
      "Processing document 24750\n",
      "Processing document 25000\n",
      "Processing document 25250\n",
      "Processing document 25500\n",
      "Processing document 25750\n",
      "Processing document 26000\n",
      "Processing document 26250\n",
      "Processing document 26500\n",
      "Processing document 26750\n",
      "Processing document 27000\n",
      "Processing document 27250\n",
      "Processing document 27500\n",
      "Processing document 27750\n",
      "Processing document 28000\n",
      "Processing document 28250\n",
      "Processing document 28500\n",
      "Processing document 28750\n",
      "Processing document 29000\n",
      "Processing document 29250\n",
      "Processing document 29500\n",
      "Processing document 29750\n",
      "Processing document 30000\n",
      "Processing document 30250\n",
      "Processing document 30500\n",
      "Processing document 30750\n",
      "Processing document 31000\n",
      "Processing document 31250\n",
      "Processing document 31500\n",
      "Processing document 31750\n",
      "Processing document 32000\n",
      "Processing document 32250\n",
      "Processing document 32500\n",
      "Processing document 32750\n",
      "Processing document 33000\n",
      "Processing document 33250\n",
      "Processing document 33500\n",
      "Processing document 33750\n",
      "Processing document 34000\n",
      "Processing document 34250\n",
      "Processing document 34500\n",
      "Processing document 34750\n",
      "Processing document 35000\n",
      "Processing document 35250\n",
      "Processing document 35500\n",
      "Processing document 35750\n",
      "Processing document 36000\n",
      "Processing document 36250\n",
      "Processing document 36500\n",
      "Processing document 36750\n",
      "Processing document 37000\n",
      "Processing document 37250\n",
      "Processing document 37500\n",
      "Processing document 37750\n",
      "Processing document 38000\n",
      "Processing document 38250\n",
      "Processing document 38500\n",
      "Processing document 38750\n",
      "Processing document 39000\n",
      "Processing document 39250\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(39319, [])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import requests\n",
    "import gzip\n",
    "import shutil\n",
    "import csv\n",
    "\n",
    "# Download the CSV file\n",
    "url = \"https://data.insideairbnb.com/united-states/ny/new-york-city/2024-03-07/data/listings.csv.gz\"\n",
    "response = requests.get(url, stream=True)\n",
    "\n",
    "# Save the downloaded file\n",
    "with open(\"listings.csv.gz\", \"wb\") as file:\n",
    "    shutil.copyfileobj(response.raw, file)\n",
    "\n",
    "# Unpack the CSV file\n",
    "with gzip.open(\"listings.csv.gz\", \"rb\") as file_in:\n",
    "    with open(\"listings.csv\", \"wb\") as file_out:\n",
    "        shutil.copyfileobj(file_in, file_out)\n",
    "\n",
    "\n",
    "def remove_empty_fields(data):\n",
    "    empty_fields = []\n",
    "    # Iterate over the dictionary items\n",
    "    for key, value in data.items():\n",
    "        # Check if the value is empty (None, empty string, empty list, etc.)\n",
    "        if not value:\n",
    "            empty_fields.append(key)\n",
    "    # Remove empty fields from the dictionary\n",
    "    for key in empty_fields:\n",
    "        del data[key]\n",
    "    return data\n",
    "\n",
    "\n",
    "def prepare_documents():\n",
    "    with open(\"listings.csv\", \"r\", encoding=\"utf-8\") as file:\n",
    "        reader = csv.DictReader(file, delimiter=\",\")\n",
    "        # we are going to only add the first 5.000 listings.\n",
    "        limit = 5000\n",
    "        for index, row in enumerate(reader):\n",
    "            if index == limit:\n",
    "                break\n",
    "            if index % 250 == 0:\n",
    "                print(f\"Processing document {index}\")\n",
    "            row[\"location\"] = {\n",
    "                \"lat\": float(row[\"latitude\"]),\n",
    "                \"lon\": float(row[\"longitude\"]),\n",
    "            }\n",
    "            row = remove_empty_fields(row)\n",
    "            yield {\n",
    "                \"_index\": \"airbnb-listings\",\n",
    "                \"_source\": dict(row),\n",
    "            }\n",
    "\n",
    "\n",
    "helpers.bulk(client, prepare_documents(), chunk_size=150)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare the MTA subway stations index\n",
    "\n",
    "We need to prepare the index and make sure that we treat the geo location as a geo location."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ObjectApiResponse({'acknowledged': True, 'shards_acknowledged': True, 'index': 'mta-stations'})"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.indices.delete(index=\"mta-stations\", ignore_unavailable=True)\n",
    "client.indices.create(\n",
    "    index=\"mta-stations\",\n",
    "    mappings={\n",
    "        \"dynamic_templates\": [\n",
    "            {\n",
    "                \"stringsaskeywords\": {\n",
    "                    \"match\": \"*\",\n",
    "                    \"match_mapping_type\": \"string\",\n",
    "                    \"mapping\": {\"type\": \"keyword\"},\n",
    "                }\n",
    "            }\n",
    "        ],\n",
    "        \"properties\": {\"location\": {\"type\": \"geo_point\"}},\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Index the MTA data\n",
    "\n",
    "We now need to index the data for the MTA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(496, [])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import csv\n",
    "\n",
    "# Download the CSV file\n",
    "url = \"https://data.ny.gov/api/views/39hk-dx4f/rows.csv?accessType=DOWNLOAD\"\n",
    "response = requests.get(url)\n",
    "\n",
    "\n",
    "# Parse and index the CSV data\n",
    "def prepare_documents():\n",
    "    reader = csv.DictReader(response.text.splitlines())\n",
    "    for row in reader:\n",
    "        row[\"location\"] = {\n",
    "            \"lat\": float(row[\"GTFS Latitude\"]),\n",
    "            \"lon\": float(row[\"GTFS Longitude\"]),\n",
    "        }\n",
    "        yield {\n",
    "            \"_index\": \"mta-stations\",\n",
    "            \"_source\": dict(row),\n",
    "        }\n",
    "\n",
    "\n",
    "# Index the documents\n",
    "helpers.bulk(client, prepare_documents())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare points of interest\n",
    "\n",
    "Same as before. We want to index the points of interests and use ELSER to make sure that any semantic searches are working. E.g. searching for `sights with gardens` should return `Central Park` even though it does not contain `garden` in the name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ObjectApiResponse({'acknowledged': True, 'shards_acknowledged': True, 'index': 'points-of-interest'})"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.indices.delete(index=\"points-of-interest\", ignore_unavailable=True)\n",
    "client.indices.create(\n",
    "    index=\"points-of-interest\",\n",
    "    settings={\"index\": {\"default_pipeline\": \"elser-ingest-pipeline\"}},\n",
    "    mappings={\n",
    "        \"dynamic_templates\": [\n",
    "            {\n",
    "                \"stringsaskeywords\": {\n",
    "                    \"match\": \"*\",\n",
    "                    \"match_mapping_type\": \"string\",\n",
    "                    \"mapping\": {\"type\": \"keyword\"},\n",
    "                }\n",
    "            }\n",
    "        ],\n",
    "        \"properties\": {\n",
    "            \"NAME\": {\"type\": \"text\"},\n",
    "            \"location\": {\"type\": \"geo_point\"},\n",
    "            \"name_embedding\": {\"type\": \"sparse_vector\"},\n",
    "        },\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download points of interest\n",
    "\n",
    "The `the_geom` looks like this: `POINT (-74.00701717096757 40.724634757833414)` which is formatted as a Well-Known Text point format and we officially support this. I personally always like to store my coordinates in lat & lon as an Object to make sure that there are no confusions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(20481, [])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import csv\n",
    "\n",
    "# Download the CSV file\n",
    "url = \"https://data.cityofnewyork.us/api/views/t95h-5fsr/rows.csv?accessType=DOWNLOAD\"\n",
    "response = requests.get(url)\n",
    "\n",
    "\n",
    "# Parse and index the CSV data\n",
    "def prepare_documents():\n",
    "    reader = csv.DictReader(response.text.splitlines())\n",
    "    for row in reader:\n",
    "        row[\"location\"] = {\n",
    "            \"lat\": float(row[\"the_geom\"].split(\" \")[2].replace(\")\", \"\")),\n",
    "            \"lon\": float(row[\"the_geom\"].split(\" \")[1].replace(\"(\", \"\")),\n",
    "        }\n",
    "        row[\"name\"] = row[\"NAME\"].lower()\n",
    "        yield {\n",
    "            \"_index\": \"points-of-interest\",\n",
    "            \"_source\": dict(row),\n",
    "        }\n",
    "\n",
    "\n",
    "# Index the documents\n",
    "helpers.bulk(client, prepare_documents())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now we have everything prepped\n",
    "\n",
    "First let's see how well ELSER does with a \"geo\" query. Let's as it for an AirBnB next to Central Park and Empire State Building. Also we are just looking at the description, not the name or the about author as of now. Let's keep it simple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 20.003891\n",
      "Title: Gorgeous 1 Bedroom - Upper East Side Manhattan -\n",
      "Location: {'lon': -73.95856, 'lat': 40.76701}\n",
      "Document ID: 3kn2hY8BHToGwgcU3erj\n",
      "\n",
      "Score: 19.757303\n",
      "Title: ~Lovely 4 Bedroom ~ Midtown ~ Grand Central~\n",
      "Location: {'lon': -73.98293292692072, 'lat': 40.74578747448903}\n",
      "Document ID: dknphY8BHToGwgcU-M6A\n",
      "\n",
      "Score: 19.757303\n",
      "Title: Lovely 4 bedrooms~Midtown grand central ~Park ave\n",
      "Location: {'lon': -73.98119, 'lat': 40.74593}\n",
      "Document ID: V0nxhY8BHToGwgcUWd78\n",
      "\n",
      "Score: 19.71977\n",
      "Title: The Center of Manhattan (West 57th Street) Apt3A\n",
      "Location: {'lon': -73.9880189, 'lat': 40.7692096}\n",
      "Document ID: n0nIhY8BHToGwgcU_IVZ\n",
      "\n",
      "Score: 19.681244\n",
      "Title: Lovely and spacious studio @Park Avenue (Midtown)\n",
      "Location: {'lon': -73.97904, 'lat': 40.74853}\n",
      "Document ID: 5knehY8BHToGwgcUmLen\n",
      "\n",
      "Score: 19.443813\n",
      "Title: Sunny Bedroom in quirky apartment\n",
      "Location: {'lon': -73.98198, 'lat': 40.74792}\n",
      "Document ID: l0n2hY8BHToGwgcUZum-\n",
      "\n",
      "Score: 19.380022\n",
      "Title: Midtown Manhattan Apt\n",
      "Location: {'lon': -73.98094, 'lat': 40.75053}\n",
      "Document ID: tUn4hY8BHToGwgcU1u85\n",
      "\n",
      "Score: 19.33765\n",
      "Title: Spacious 4BR ~ Empire ST ~ Grand Central~ Park Ave\n",
      "Location: {'lon': -73.98324545084193, 'lat': 40.745539903474224}\n",
      "Document ID: tUnrhY8BHToGwgcUFtDL\n",
      "\n",
      "Score: 19.33765\n",
      "Title: *NEW* Spacious 4BR ~Park Ave~Empire ST~Murray Hill\n",
      "Location: {'lon': -73.98241, 'lat': 40.74386}\n",
      "Document ID: 8UnxhY8BHToGwgcUWd38\n",
      "\n",
      "Score: 19.135761\n",
      "Title: Park Ave Apt in the Heart of NYC\n",
      "Location: {'lon': -73.97944, 'lat': 40.75082}\n",
      "Document ID: 8knzhY8BHToGwgcU4ePP\n",
      "\n"
     ]
    }
   ],
   "source": [
    "response = client.search(\n",
    "    index=\"airbnb-*\",\n",
    "    size=10,\n",
    "    query={\n",
    "        \"text_expansion\": {\n",
    "            \"description_embedding\": {\n",
    "                \"model_id\": \".elser_model_2_linux-x86_64\",\n",
    "                \"model_text\": \"Next to Central Park and Empire State Building\",\n",
    "            }\n",
    "        }\n",
    "    },\n",
    ")\n",
    "\n",
    "for hit in response[\"hits\"][\"hits\"]:\n",
    "    doc_id = hit[\"_id\"]\n",
    "    score = hit[\"_score\"]\n",
    "    name = hit[\"_source\"][\"name\"]\n",
    "    location = hit[\"_source\"][\"location\"]\n",
    "    print(\n",
    "        f\"Score: {score}\\nTitle: {name}\\nLocation: {location}\\nDocument ID: {doc_id}\\n\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analysing the response\n",
    "\n",
    "We indexed all AirBnBs, so it might be little different to what you get when you only do the first 5.000.\n",
    "\n",
    "The next step is to run a geo_distance query within Elasticsearch. First to analyse how far apart `Central Park` and `Empire State Building` is. Since the `Central Park` is pretty big and contains a multitude of points of interest, we will use the `Bow Bridge` an iconic sight.\n",
    "\n",
    "We will use a simple terms query to get the geo location of `Central Park Bow Bridge` and then run a `geo_distance` query with a `_geo_distance` sort to get the exact distance back. The `geo_distance` query as of now always requires a `distance` parameter. We add a `term` to search for `empire state building` since we are just interested in this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: central park bow bridge\n",
      "Location: {'lon': -73.97178440451849, 'lat': 40.77577539823907}\n",
      "\n",
      "Distance to Empire State Building: 3.247504472145157 km\n"
     ]
    }
   ],
   "source": [
    "response = client.search(\n",
    "    index=\"points-of-interest\",\n",
    "    size=1,\n",
    "    query={\"term\": {\"name\": \"central park bow bridge\"}},\n",
    ")\n",
    "\n",
    "for hit in response[\"hits\"][\"hits\"]:\n",
    "    # this should now be the central park bow bridge.\n",
    "    print(f\"Name: {hit['_source']['name']}\\nLocation: {hit['_source']['location']}\\n\")\n",
    "    response = client.search(\n",
    "        index=\"points-of-interest\",\n",
    "        size=1,\n",
    "        query={\n",
    "            \"bool\": {\n",
    "                \"must\": {\"term\": {\"name\": \"empire state building\"}},\n",
    "                \"filter\": {\n",
    "                    \"geo_distance\": {\n",
    "                        \"distance\": \"200km\",\n",
    "                        \"location\": {\n",
    "                            \"lat\": hit[\"_source\"][\"location\"][\"lat\"],\n",
    "                            \"lon\": hit[\"_source\"][\"location\"][\"lon\"],\n",
    "                        },\n",
    "                    }\n",
    "                },\n",
    "            }\n",
    "        },\n",
    "        sort=[\n",
    "            {\n",
    "                \"_geo_distance\": {\n",
    "                    \"location\": {\n",
    "                        \"lat\": hit[\"_source\"][\"location\"][\"lat\"],\n",
    "                        \"lon\": hit[\"_source\"][\"location\"][\"lon\"],\n",
    "                    },\n",
    "                    \"unit\": \"km\",\n",
    "                    \"distance_type\": \"plane\",\n",
    "                    \"order\": \"asc\",\n",
    "                }\n",
    "            }\n",
    "        ],\n",
    "    )\n",
    "    print(\n",
    "        f\"Distance to Empire State Building: {response['hits']['hits'][0]['sort'][0]} km\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparing to Elser\n",
    "\n",
    "Now our top scoring document:\n",
    "\n",
    "```\n",
    "Score: 20.003891\n",
    "Title: Gorgeous 1 Bedroom - Upper East Side Manhattan -\n",
    "Location: {'lon': -73.95856, 'lat': 40.76701}\n",
    "Document ID: AkgfEI8BHToGwgcUA6-7\n",
    "```\n",
    "\n",
    "Let's run the calculation from above using geo_distance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distance between AirBnB and central park bow bridge 1.4799179352060348 km\n",
      "Distance between AirBnB and empire state building 3.0577584374128617 km\n"
     ]
    }
   ],
   "source": [
    "response = client.search(\n",
    "    index=\"points-of-interest\",\n",
    "    size=10,\n",
    "    query={\n",
    "        \"bool\": {\n",
    "            \"must\": {\n",
    "                \"terms\": {\"name\": [\"central park bow bridge\", \"empire state building\"]}\n",
    "            },\n",
    "            \"filter\": {\n",
    "                \"geo_distance\": {\n",
    "                    \"distance\": \"200km\",\n",
    "                    \"location\": {\"lat\": \"40.76701\", \"lon\": \"-73.95856\"},\n",
    "                }\n",
    "            },\n",
    "        }\n",
    "    },\n",
    "    sort=[\n",
    "        {\n",
    "            \"_geo_distance\": {\n",
    "                \"location\": {\"lat\": \"40.76701\", \"lon\": \"-73.95856\"},\n",
    "                \"unit\": \"km\",\n",
    "                \"distance_type\": \"plane\",\n",
    "                \"order\": \"asc\",\n",
    "            }\n",
    "        }\n",
    "    ],\n",
    ")\n",
    "\n",
    "for hit in response[\"hits\"][\"hits\"]:\n",
    "    print(\"Distance between AirBnB and\", hit[\"_source\"][\"name\"], hit[\"sort\"][0], \"km\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analysing\n",
    "\n",
    "Only 1.4km and 3km away from the two sights. Not that bad. Let's see what we can find when we create a geo-bounding box with the Empire State Building and the Central Park Bow Bridge. Additionally we will sort the result by the distance to the Central Park Bow Bridge and then by distance to Empire State Building."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distance to Central Park Bow Bridge: 0.4131817369844223 km\n",
      "Distance to Empire State Building: 3.089519162860651 km\n",
      "Title: Great Room by Central Park & Time Square\n",
      "Document ID: 80n9hY8BHToGwgcUUfrA\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.45350835952906815 km\n",
      "Distance to Empire State Building: 3.1145514127463594 km\n",
      "Title: 5 Beds, Steps from Lincoln Center, Central Park\n",
      "Document ID: 8UnOhY8BHToGwgcUTJKv\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.5671263759837657 km\n",
      "Distance to Empire State Building: 2.955994026017429 km\n",
      "Title: Huge 1 Bedroom Entire Unit on Central Park! UWS\n",
      "Document ID: F0nThY8BHToGwgcUbJ95\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.5952177487694544 km\n",
      "Distance to Empire State Building: 3.0537706667049194 km\n",
      "Title: Bright Comfy Studio!\n",
      "Document ID: kknThY8BHToGwgcU-6D0\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.5997105192552462 km\n",
      "Distance to Empire State Building: 2.8883161884323183 km\n",
      "Title: Large duplex apartment Lincoln Sq\n",
      "Document ID: nEn1hY8BHToGwgcUbefs\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.6145766439818324 km\n",
      "Distance to Empire State Building: 2.964903765549427 km\n",
      "Title: * Light & Bright Central Park West Brownstone\n",
      "Document ID: ZEnthY8BHToGwgcUWtXB\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.6179705737259344 km\n",
      "Distance to Empire State Building: 3.0671730371540322 km\n",
      "Title: Lovely 4 Bedroom/3 Bathroom/Garden. Central Park.\n",
      "Document ID: FknVhY8BHToGwgcUCqMx\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.6651772103309174 km\n",
      "Distance to Empire State Building: 2.9487596701698107 km\n",
      "Title: Stylish Cozy Entire Studio\n",
      "Document ID: V0nIhY8BHToGwgcU_IVZ\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.6806723256826838 km\n",
      "Distance to Empire State Building: 2.953411336952507 km\n",
      "Title: The Sunny Lincoln Center Apartment\n",
      "Document ID: 3EnVhY8BHToGwgcUmKMH\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.7233438162417364 km\n",
      "Distance to Empire State Building: 2.9758539550319947 km\n",
      "Title: Lincoln Center Gem - Short Term Available\n",
      "Document ID: GEnJhY8BHToGwgcUM4Zy\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.7443571694818886 km\n",
      "Distance to Empire State Building: 3.0437869095436754 km\n",
      "Title: Charming Studio UWS near Central Park\n",
      "Document ID: U0oBho8BHToGwgcU8gWr\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.7522774827930032 km\n",
      "Distance to Empire State Building: 2.788981551632797 km\n",
      "Title: Shared room at Lincoln Center\n",
      "Document ID: PEnGhY8BHToGwgcUk3-t\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.7613822055498712 km\n",
      "Distance to Empire State Building: 2.917809424236674 km\n",
      "Title: Cozy and charming NYC studio\n",
      "Document ID: sknxhY8BHToGwgcUFN0Q\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.7655012840614352 km\n",
      "Distance to Empire State Building: 3.0378129044232347 km\n",
      "Title: 1 Bedroom Apt, West 69th St, bwtn BWay & Columbus\n",
      "Document ID: W0nWhY8BHToGwgcUl6ZH\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.7733958540516488 km\n",
      "Distance to Empire State Building: 2.7720609897741144 km\n",
      "Title: Perfectly Located Lincoln Center One-Bedroom\n",
      "Document ID: 2EnThY8BHToGwgcUsJ_H\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.7863631036930467 km\n",
      "Distance to Empire State Building: 2.917713385537012 km\n",
      "Title: Apt. across from Lincoln Center\n",
      "Document ID: _Un5hY8BHToGwgcUzPHd\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.8285887310207944 km\n",
      "Distance to Empire State Building: 2.654302842845152 km\n",
      "Title: Blueground | UWS, w/d, nr Lincon Center\n",
      "Document ID: F0nBhY8BHToGwgcU4HA_\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.8285887310207944 km\n",
      "Distance to Empire State Building: 2.654302842845152 km\n",
      "Title: Blueground | UWS, doorman, nr central park\n",
      "Document ID: 0UnqhY8BHToGwgcUQM7z\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.8317793813857818 km\n",
      "Distance to Empire State Building: 2.989653305906252 km\n",
      "Title: Large 1 bdrm apt, 1 block to central pk 30 DAY MIN\n",
      "Document ID: 9UnxhY8BHToGwgcUWd38\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.8450353341336536 km\n",
      "Distance to Empire State Building: 2.990493364568853 km\n",
      "Title: WONDERFUL, COMFORTABLE, SAFE LINCOLN CENTER SPACE\n",
      "Document ID: jUnyhY8BHToGwgcUX-D6\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.850150844706325 km\n",
      "Distance to Empire State Building: 2.7935321567649516 km\n",
      "Title: Blueground | UWS, doorman, elevator, nr park\n",
      "Document ID: 8EnihY8BHToGwgcUPr5p\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.850317121579706 km\n",
      "Distance to Empire State Building: 2.793555713498908 km\n",
      "Title: Blueground | UWS, doorman, nr Lincoln Center\n",
      "Document ID: rknxhY8BHToGwgcUFN0Q\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.850317121579706 km\n",
      "Distance to Empire State Building: 2.793555713498908 km\n",
      "Title: Blueground | UWS, doorman, elevator, nr park\n",
      "Document ID: D0nhhY8BHToGwgcUU70s\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.8607277425960257 km\n",
      "Distance to Empire State Building: 2.855736774975596 km\n",
      "Title: Cute 2BR by Lincoln Square\n",
      "Document ID: cUnfhY8BHToGwgcUKbm8\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.8849253434847821 km\n",
      "Distance to Empire State Building: 2.6888159069336384 km\n",
      "Title: Sunny room beside Central Park\n",
      "Document ID: BEnehY8BHToGwgcUEbeg\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.8899297974196853 km\n",
      "Distance to Empire State Building: 2.593498488090597 km\n",
      "Title: Lincoln Center  luxury condo\n",
      "Document ID: 9En1hY8BHToGwgcUr-eV\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.9089851862111872 km\n",
      "Distance to Empire State Building: 2.756559907255608 km\n",
      "Title: Writer's Retreat in the Heart of the City\n",
      "Document ID: SUn9hY8BHToGwgcUmPvo\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.9179057084098066 km\n",
      "Distance to Empire State Building: 2.633649076174077 km\n",
      "Title: Manhattan New York Upper West side\n",
      "Document ID: LUnChY8BHToGwgcUjnIE\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.9376523264959093 km\n",
      "Distance to Empire State Building: 2.5614219378831278 km\n",
      "Title: Literally steps from Central Park!\n",
      "Document ID: Ukn8hY8BHToGwgcUm_kq\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.9445686078004821 km\n",
      "Distance to Empire State Building: 3.0297909425761445 km\n",
      "Title: Sunny Studio Near Central Park\n",
      "Document ID: VUn5hY8BHToGwgcUjvHq\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.9642705205066279 km\n",
      "Distance to Empire State Building: 2.9953515473116803 km\n",
      "Title: one bedroom apt for long term stay up to 2 months\n",
      "Document ID: ZEnYhY8BHToGwgcUZ6qt\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.9722789279214233 km\n",
      "Distance to Empire State Building: 2.539286718831572 km\n",
      "Title: Stunning view - Columbus Circle area\n",
      "Document ID: -knThY8BHToGwgcUKJ4Z\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.9787646884032914 km\n",
      "Distance to Empire State Building: 2.603929390209796 km\n",
      "Title: Peaceful Central Park & Lincoln Center Home\n",
      "Document ID: xkn6hY8BHToGwgcUdfMU\n",
      "\n",
      "Distance to Central Park Bow Bridge: 0.9871880696323352 km\n",
      "Distance to Empire State Building: 2.610369090495495 km\n",
      "Title: PRIVATE ENTIRE FLOOR -NEXT TO Central PARK  !\n",
      "Document ID: 10nthY8BHToGwgcUCdS_\n",
      "\n",
      "Distance to Central Park Bow Bridge: 1.0178197740885386 km\n",
      "Distance to Empire State Building: 2.6488315240312748 km\n",
      "Title: Central Park / UWS\n",
      "Document ID: TknvhY8BHToGwgcUW9n4\n",
      "\n",
      "Distance to Central Park Bow Bridge: 1.0265048444154021 km\n",
      "Distance to Empire State Building: 2.4909589840010624 km\n",
      "Title: Best views of Manhattan\n",
      "Document ID: HUnVhY8BHToGwgcUmKQH\n",
      "\n",
      "Distance to Central Park Bow Bridge: 1.0451374806451885 km\n",
      "Distance to Empire State Building: 2.523276699341922 km\n",
      "Title: Spectacular Lincoln Center Condo in the sky\n",
      "Document ID: sknrhY8BHToGwgcUqNF5\n",
      "\n",
      "Distance to Central Park Bow Bridge: 1.047774729331466 km\n",
      "Distance to Empire State Building: 2.561250086148971 km\n",
      "Title: Gorgeous Apt. on Central Park\n",
      "Document ID: -knuhY8BHToGwgcUNtZz\n",
      "\n",
      "Distance to Central Park Bow Bridge: 1.0579310265078672 km\n",
      "Distance to Empire State Building: 2.477170281896128 km\n",
      "Title: Lincoln Center Luxury High Rise 1 bedroom\n",
      "Document ID: 9EnehY8BHToGwgcUmLen\n",
      "\n",
      "Distance to Central Park Bow Bridge: 1.0600173189705104 km\n",
      "Distance to Empire State Building: 2.5544718006352745 km\n",
      "Title: Take it now you won't find better\n",
      "Document ID: fknwhY8BHToGwgcUU9v3\n",
      "\n",
      "Distance to Central Park Bow Bridge: 1.0674248087467735 km\n",
      "Distance to Empire State Building: 2.3463319833393945 km\n",
      "Title: Experience paradise on earth in midtown\n",
      "Document ID: 8En9hY8BHToGwgcUFfkh\n",
      "\n",
      "Distance to Central Park Bow Bridge: 1.068922155759417 km\n",
      "Distance to Empire State Building: 2.969897996609374 km\n",
      "Title: An amazing 1 bdr apt next to The Met opera.\n",
      "Document ID: c0nZhY8BHToGwgcUMaym\n",
      "\n",
      "Distance to Central Park Bow Bridge: 1.0724820481304276 km\n",
      "Distance to Empire State Building: 3.0238311506341047 km\n",
      "Title: Beautiful 1-bedroom in Lincoln Sqr- gorgeous view\n",
      "Document ID: mEnUhY8BHToGwgcUvaLK\n",
      "\n",
      "Distance to Central Park Bow Bridge: 1.106649499174859 km\n",
      "Distance to Empire State Building: 2.605055542768263 km\n",
      "Title: Skyline view apt -Lincoln Square\n",
      "Document ID: 7knxhY8BHToGwgcUWd38\n",
      "\n",
      "Distance to Central Park Bow Bridge: 1.1240321000024536 km\n",
      "Distance to Empire State Building: 2.333216200760106 km\n",
      "Title: Times Square, Central Park, NYC, Luxury, Summer\n",
      "Document ID: fknJhY8BHToGwgcUkYfd\n",
      "\n",
      "Distance to Central Park Bow Bridge: 1.1243476248351183 km\n",
      "Distance to Empire State Building: 2.4285558932557834 km\n",
      "Title: Luxury 3 BR condo + balcony w views of Central PK\n",
      "Document ID: BknBhY8BHToGwgcU4HA_\n",
      "\n",
      "Distance to Central Park Bow Bridge: 1.1526079295801053 km\n",
      "Distance to Empire State Building: 2.202056866206072 km\n",
      "Title: Courtyard View Room in Manhattan\n",
      "Document ID: tEnYhY8BHToGwgcUK6nx\n",
      "\n",
      "Distance to Central Park Bow Bridge: 1.164720035883416 km\n",
      "Distance to Empire State Building: 2.098650122275327 km\n",
      "Title: Gorgeous view of the Hudson River\n",
      "Document ID: FEnNhY8BHToGwgcU8ZKo\n",
      "\n",
      "Distance to Central Park Bow Bridge: 1.1712213104190234 km\n",
      "Distance to Empire State Building: 2.2972489327173635 km\n",
      "Title: Upper West Apartment Block Away from Central Park!\n",
      "Document ID: _knGhY8BHToGwgcUk36t\n",
      "\n",
      "Distance to Central Park Bow Bridge: 1.174417314727233 km\n",
      "Distance to Empire State Building: 2.0828398365159138 km\n",
      "Title: King Suite with Central Park Views\n",
      "Document ID: oUnThY8BHToGwgcUKJ4Z\n",
      "\n"
     ]
    }
   ],
   "source": [
    "response = client.search(\n",
    "    index=\"points-of-interest\",\n",
    "    size=2,\n",
    "    query={\"terms\": {\"name\": [\"central park bow bridge\", \"empire state building\"]}},\n",
    ")\n",
    "\n",
    "# for easier access we store the locations in two variables\n",
    "central = {}\n",
    "empire = {}\n",
    "for hit in response[\"hits\"][\"hits\"]:\n",
    "    hit = hit[\"_source\"]\n",
    "    if \"central park bow bridge\" in hit[\"name\"]:\n",
    "        central = hit[\"location\"]\n",
    "    elif \"empire state building\" in hit[\"name\"]:\n",
    "        empire = hit[\"location\"]\n",
    "\n",
    "# Now we can run the geo_bounding_box query and sort it by the\n",
    "# distance first to Central Park Bow Bridge\n",
    "# and then to the Empire State Building.\n",
    "response = client.search(\n",
    "    index=\"airbnb-*\",\n",
    "    size=50,\n",
    "    query={\n",
    "        \"geo_bounding_box\": {\n",
    "            \"location\": {\n",
    "                \"top_left\": {\"lat\": central[\"lat\"], \"lon\": empire[\"lon\"]},\n",
    "                \"bottom_right\": {\"lat\": empire[\"lat\"], \"lon\": central[\"lon\"]},\n",
    "            }\n",
    "        }\n",
    "    },\n",
    "    sort=[\n",
    "        {\n",
    "            \"_geo_distance\": {\n",
    "                \"location\": {\"lat\": central[\"lat\"], \"lon\": central[\"lon\"]},\n",
    "                \"unit\": \"km\",\n",
    "                \"distance_type\": \"plane\",\n",
    "                \"order\": \"asc\",\n",
    "            }\n",
    "        },\n",
    "        {\n",
    "            \"_geo_distance\": {\n",
    "                \"location\": {\"lat\": empire[\"lat\"], \"lon\": empire[\"lon\"]},\n",
    "                \"unit\": \"km\",\n",
    "                \"distance_type\": \"plane\",\n",
    "                \"order\": \"asc\",\n",
    "            }\n",
    "        },\n",
    "    ],\n",
    ")\n",
    "\n",
    "for hit in response[\"hits\"][\"hits\"]:\n",
    "    print(f\"Distance to Central Park Bow Bridge: {hit['sort'][0]} km\")\n",
    "    print(f\"Distance to Empire State Building: {hit['sort'][1]} km\")\n",
    "    print(f\"Title: {hit['_source']['name']}\\nDocument ID: {hit['_id']}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AI\n",
    "\n",
    "Now let's finally get to the AI part. All of this was the setup and understanding what makes geo spatial searches tick and how they work. There is still a lot more to discover. Let's hookup it up to our OpenAI instance. In here we use the Azure OpenAI resource."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We are using the Azure OpenAI Resource.\n",
    "OPENAI_API_KEY = getpass(\"OpenAI API key: \")\n",
    "\n",
    "# Set API key\n",
    "oai_client = AzureOpenAI(\n",
    "    # This is the default and can be omitted\n",
    "    api_key=OPENAI_API_KEY,\n",
    "    api_version=\"2024-02-01\",\n",
    "    azure_endpoint=getpass(\"OpenAI Endpoint: \"),\n",
    ")\n",
    "# Define model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\": \"Paris*\", //\n"
     ]
    }
   ],
   "source": [
    "# Let's define the model we are using\n",
    "model = \"gpt-35-turbo\"\n",
    "\n",
    "# Let's do a test:\n",
    "question = \"What is the capital of France? Answer with just the capital city.\"\n",
    "\n",
    "# Create test question.\n",
    "# We don't really care about what we are asking. We just want to verify that the connection is working.\n",
    "answer = oai_client.completions.create(prompt=question, model=model, max_tokens=5)\n",
    "print(answer.choices[0].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that this works, we are sure that we are in the correct place to start our question. We are writing a prompt that forces ChatGPT to create a JSON response and extract the information from the question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "```\n",
      "{\n",
      "  \"what\": \"AirBnB\",\n",
      "  \"near\": \"Empire State Building\",\n",
      "  \"distance_in_km\": 1.609344\n",
      "}\n",
      "```\n",
      "Here is some sample code:\n",
      "```\n",
      "from pprint import pprint\n",
      "sentence = \"Get me the closest AirBnB between 1 miles distance from the Empire State Building\"\n",
      "pprint(nlbot.get_what_near_distance(sentence))\n",
      "```\n",
      "kpps_textex 2021-07-07: This\n"
     ]
    }
   ],
   "source": [
    "question = \"\"\"\n",
    "As an expert in named entity recognition machine learning models, I will give you a sentence from which I would like you to extract what needs to be found (location, apartment, airbnb, sight, etc) near which location and the distance between them. The distance needs to be a number expressed in kilometers. I would like the result to be expressed in JSON with the following fields: \"what\", \"near\", \"distance_in_km\". Only return the JSON.\n",
    "Here is the sentence: \"Get me the closest AirBnB between 1 miles distance from the Empire State Building\"\n",
    "\"\"\"\n",
    "\n",
    "answer = oai_client.completions.create(prompt=question, model=model, max_tokens=100)\n",
    "print(answer.choices[0].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The answer in our case is the following\n",
    "\n",
    "Here is the desired output:\n",
    "```\n",
    "{\n",
    "    \"what\": \"AirBnB\",\n",
    "    \"near\": \"Empire State Building\",\n",
    "    \"distance_in_km\": 1610\n",
    "}\n",
    "```\n",
    "1) Extract distance - done (1 miles)\n",
    "2) Convert distance to km - done (1.6 km)\n",
    "3) Extract location - This should be \"Empire State Building\", but in more general terms we should recognize that this is a location so we make a separate label called"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distance to Empire State Building: 0.004002111094864837 km\n",
      "Title: Comfort and Convenience! 2 Units Near Bryant Park!\n",
      "Document ID: DUn3hY8BHToGwgcUWuz5\n",
      "\n",
      "Distance to Empire State Building: 0.011231615140053008 km\n",
      "Title: Relax and Recharge! 3 Relaxing Units, Pets Allowed\n",
      "Document ID: O0n3hY8BHToGwgcUWuz5\n",
      "\n",
      "Distance to Empire State Building: 0.02349319910825568 km\n",
      "Title: Quiet space.\n",
      "Document ID: EEnohY8BHToGwgcUXMtf\n",
      "\n",
      "Distance to Empire State Building: 0.027646218010495525 km\n",
      "Title: 4 Serene Units In The Heart Of NYC! Pets Allowed!\n",
      "Document ID: QEn2hY8BHToGwgcU3evj\n",
      "\n",
      "Distance to Empire State Building: 0.033996880421693895 km\n",
      "Title: Prime Location! 3 Relaxing Units, Pets Allowed\n",
      "Document ID: okn3hY8BHToGwgcUmux7\n",
      "\n",
      "Distance to Empire State Building: 0.03531856438046657 km\n",
      "Title: Visit and Catch A Show On Broadway! 4 Deluxe Units\n",
      "Document ID: lkn2hY8BHToGwgcUnOqR\n",
      "\n",
      "Distance to Empire State Building: 0.0374488409537633 km\n",
      "Title: City Escape! Unit Close to Madison Square Garden!\n",
      "Document ID: oEn2hY8BHToGwgcUnOqR\n",
      "\n",
      "Distance to Empire State Building: 0.037660236640884664 km\n",
      "Title: Three Pet-friendly Units, Minutes from MoMA!\n",
      "Document ID: qkn3hY8BHToGwgcUmux7\n",
      "\n",
      "Distance to Empire State Building: 0.03987374959359285 km\n",
      "Title: Private Luxury Room In Manhattan, New York City!\n",
      "Document ID: tEnQhY8BHToGwgcUPJgZ\n",
      "\n",
      "Distance to Empire State Building: 0.03988503702305329 km\n",
      "Title: The Gregory Hotel, Tailored King\n",
      "Document ID: 4EnyhY8BHToGwgcUn-AZ\n",
      "\n",
      "Distance to Empire State Building: 0.0409575446637707 km\n",
      "Title: Cozy Clean Room in Manhattan 32nd & 5th/Madison\n",
      "Document ID: 3knyhY8BHToGwgcU2eFr\n",
      "\n",
      "Distance to Empire State Building: 0.04197878171563393 km\n",
      "Title: Prime Location! 2 Units Near Rockefeller Center!\n",
      "Document ID: 2kn3hY8BHToGwgcUmux7\n",
      "\n",
      "Distance to Empire State Building: 0.04390474835251669 km\n",
      "Title: Look No Further! Pet-friendly Property in NYC!\n",
      "Document ID: AUn2hY8BHToGwgcUZuq-\n",
      "\n",
      "Distance to Empire State Building: 0.05098654306582594 km\n",
      "Title: Prime Location! 4 Units Near Top of the Rock!\n",
      "Document ID: gEn3hY8BHToGwgcUmux7\n",
      "\n",
      "Distance to Empire State Building: 0.053455403752978005 km\n",
      "Title: Amazing Studio-EmpireStateBuilding\n",
      "Document ID: FUnbhY8BHToGwgcUubJO\n",
      "\n",
      "Distance to Empire State Building: 0.05892683320207764 km\n",
      "Title: You Found It! Minutes to Verizon Plaza\n",
      "Document ID: eUnWhY8BHToGwgcUl6ZH\n",
      "\n",
      "Distance to Empire State Building: 0.06445554302109083 km\n",
      "Title: Prime Location! Pets Allowed, Walk to Bryant Park!\n",
      "Document ID: g0n2hY8BHToGwgcUnOqR\n",
      "\n",
      "Distance to Empire State Building: 0.06902579654288031 km\n",
      "Title: 1 Bedroom Apt near the Empire State\n",
      "Document ID: V0nXhY8BHToGwgcU6anA\n",
      "\n",
      "Distance to Empire State Building: 0.07179165027056177 km\n",
      "Title: Gorgeous 1 bedroom luxury condo\n",
      "Document ID: ZUnChY8BHToGwgcUUXGa\n",
      "\n",
      "Distance to Empire State Building: 0.07241259883694023 km\n",
      "Title: Great Relaxing Stay! Two Pet-friendly Units!\n",
      "Document ID: o0r_hY8BHToGwgcU3gB9\n",
      "\n",
      "Distance to Empire State Building: 0.07734670628382642 km\n",
      "Title: Times Square centrally located large room\n",
      "Document ID: wEnPhY8BHToGwgcUr5cL\n",
      "\n",
      "Distance to Empire State Building: 0.08380964458513444 km\n",
      "Title: Unforgettable NYC Trip! Minutes from Times Square!\n",
      "Document ID: E0n0hY8BHToGwgcUY-UA\n",
      "\n",
      "Distance to Empire State Building: 0.09093210159321162 km\n",
      "Title: LARGE PRIVATE ROOM BY EVERYTHING MIDTOWN MANHATTAN\n",
      "Document ID: PknwhY8BHToGwgcUGNup\n",
      "\n",
      "Distance to Empire State Building: 0.09349715729963792 km\n",
      "Title: !!Studio next to Empire State Bldg.\n",
      "Document ID: bEn6hY8BHToGwgcUAvJ5\n",
      "\n",
      "Distance to Empire State Building: 0.09395181480762141 km\n",
      "Title: ICONIC STUDIO MIDTOWN 5TH AVENUE!\n",
      "Document ID: mknWhY8BHToGwgcUl6ZH\n",
      "\n",
      "Distance to Empire State Building: 0.09437030513272562 km\n",
      "Title: A Unique Gem in the Heart of NYC! Pets Allowed!\n",
      "Document ID: u0nehY8BHToGwgcUmLen\n",
      "\n",
      "Distance to Empire State Building: 0.09443764032765753 km\n",
      "Title: You Found it! Pet-friendly Property in NYC!\n",
      "Document ID: wUr_hY8BHToGwgcU3gB9\n",
      "\n",
      "Distance to Empire State Building: 0.09558644957518557 km\n",
      "Title: City Escape! 2 Units, Walk to Times Square!\n",
      "Document ID: I0nRhY8BHToGwgcUqZxI\n",
      "\n",
      "Distance to Empire State Building: 0.09577186077892401 km\n",
      "Title: Bright and Sunny Luxury Modern Midtown Apartment\n",
      "Document ID: jEoBho8BHToGwgcUsARr\n",
      "\n",
      "Distance to Empire State Building: 0.09854289242947946 km\n",
      "Title: Spacious Room in the heart of Manhattan\n",
      "Document ID: hUoBho8BHToGwgcUsARr\n",
      "\n",
      "Distance to Empire State Building: 0.1002689897194992 km\n",
      "Title: Look No Further! 4 Comfortable Units, Pets Allowed\n",
      "Document ID: uEn_hY8BHToGwgcUIf4h\n",
      "\n",
      "Distance to Empire State Building: 0.10410058755391106 km\n",
      "Title: 2 Comfortable Units Close to Rockefeller Center!\n",
      "Document ID: c0nRhY8BHToGwgcUY5vj\n",
      "\n",
      "Distance to Empire State Building: 0.11491975195324128 km\n",
      "Title: Private Room 717 | Shared Bath\n",
      "Document ID: HEn2hY8BHToGwgcUJukN\n",
      "\n",
      "Distance to Empire State Building: 0.11556008344777294 km\n",
      "Title: Cute Studio flat near Times Square\n",
      "Document ID: ZUn-hY8BHToGwgcUH_y4\n",
      "\n",
      "Distance to Empire State Building: 0.1165303750375657 km\n",
      "Title: Rare Gem! 2 Convenient Units, Pets Allowed\n",
      "Document ID: K0nbhY8BHToGwgcUPLF-\n",
      "\n",
      "Distance to Empire State Building: 0.11711484106305878 km\n",
      "Title: Exclusive Private Studio 1103 | Private Bathroom\n",
      "Document ID: xEnChY8BHToGwgcUyHLx\n",
      "\n",
      "Distance to Empire State Building: 0.11786577229462668 km\n",
      "Title: Room available in Midtown\n",
      "Document ID: SUnFhY8BHToGwgcUsnxM\n",
      "\n",
      "Distance to Empire State Building: 0.12034939010787557 km\n",
      "Title: Amazing City View Ultra Luxury One Bed in Midtown\n",
      "Document ID: G0oBho8BHToGwgcUcASn\n",
      "\n",
      "Distance to Empire State Building: 0.12110771866347951 km\n",
      "Title: Small Cozy Room in Herald Square\n",
      "Document ID: O0n5hY8BHToGwgcUjvHq\n",
      "\n",
      "Distance to Empire State Building: 0.12212194882486152 km\n",
      "Title: Nice Studio in Midtown Manhattan\n",
      "Document ID: v0nVhY8BHToGwgcUT6Op\n",
      "\n",
      "Distance to Empire State Building: 0.12218665438647491 km\n",
      "Title: A+ Location Deluxe Studio(3 beds) #5\n",
      "Document ID: w0nDhY8BHToGwgcUU3QL\n",
      "\n",
      "Distance to Empire State Building: 0.12492688971642316 km\n",
      "Title: Midtown Oasis - Steps from Empire State Building\n",
      "Document ID: hkn0hY8BHToGwgcUouWl\n",
      "\n",
      "Distance to Empire State Building: 0.12874431225566033 km\n",
      "Title: Family Studio With 3 Beds close Empire State #29\n",
      "Document ID: 6knwhY8BHToGwgcUztx6\n",
      "\n",
      "Distance to Empire State Building: 0.12887493094730135 km\n",
      "Title: Studio in luxury building at the Center of NYC\n",
      "Document ID: eEn8hY8BHToGwgcUWPg6\n",
      "\n",
      "Distance to Empire State Building: 0.12888385348560277 km\n",
      "Title: Relaxation Meets Convenience! Pets Allowed\n",
      "Document ID: LEn2hY8BHToGwgcU3evj\n",
      "\n",
      "Distance to Empire State Building: 0.1291062171791793 km\n",
      "Title: Exclusive Private Studio *35 | Private Bathroom\n",
      "Document ID: 1Un2hY8BHToGwgcUZum-\n",
      "\n",
      "Distance to Empire State Building: 0.13174203146474467 km\n",
      "Title: Imperial Midtown Studios 829\n",
      "Document ID: d0nwhY8BHToGwgcUkNxN\n",
      "\n",
      "Distance to Empire State Building: 0.13292008444030254 km\n",
      "Title: Studio Apartment #2\n",
      "Document ID: cEnXhY8BHToGwgcUGacW\n",
      "\n",
      "Distance to Empire State Building: 0.134363624510592 km\n",
      "Title: Cozy corner near Empire State Building\n",
      "Document ID: j0oBho8BHToGwgcUsARr\n",
      "\n",
      "Distance to Empire State Building: 0.13472984786797276 km\n",
      "Title: A+ Location Comfort Studio Apartment (3 Beds) #2\n",
      "Document ID: 7En-hY8BHToGwgcUYPyr\n",
      "\n",
      "Distance to Empire State Building: 0.1349246423105236 km\n",
      "Title: Exclusive Private Room 613 | Shared Bathroom Only\n",
      "Document ID: mkn9hY8BHToGwgcUUfrA\n",
      "\n",
      "Distance to Empire State Building: 0.13522578965449172 km\n",
      "Title: A+ Location Superior Studio (3 Beds)\n",
      "Document ID: e0nJhY8BHToGwgcUkYfd\n",
      "\n",
      "Distance to Empire State Building: 0.1356533641942277 km\n",
      "Title: Ultimate NYC Trip! Unit w/ Empire State View\n",
      "Document ID: AEn0hY8BHToGwgcUY-UA\n",
      "\n",
      "Distance to Empire State Building: 0.1361089243182925 km\n",
      "Title: Well appointed queen with ADA features\n",
      "Document ID: k0nFhY8BHToGwgcUhXt2\n",
      "\n",
      "Distance to Empire State Building: 0.1361089243182925 km\n",
      "Title: Easy access to transit for seeing all the sights\n",
      "Document ID: NEnFhY8BHToGwgcUsnxM\n",
      "\n",
      "Distance to Empire State Building: 0.1361089243182925 km\n",
      "Title: High-speed WiFi and in-room coffee maker\n",
      "Document ID: IUn0hY8BHToGwgcUY-UA\n",
      "\n",
      "Distance to Empire State Building: 0.1361089243182925 km\n",
      "Title: Incredible views from your own private terrace\n",
      "Document ID: ikn3hY8BHToGwgcUmux7\n",
      "\n",
      "Distance to Empire State Building: 0.1361089243182925 km\n",
      "Title: Bring the kiddos – there’s room!\n",
      "Document ID: 60n3hY8BHToGwgcUmux7\n",
      "\n",
      "Distance to Empire State Building: 0.1361089243182925 km\n",
      "Title: Comfortable king with accessible features\n",
      "Document ID: 9kn3hY8BHToGwgcUWuv5\n",
      "\n",
      "Distance to Empire State Building: 0.1361089243182925 km\n",
      "Title: 24/7 gym and business center\n",
      "Document ID: Akn3hY8BHToGwgcUWuz5\n",
      "\n",
      "Distance to Empire State Building: 0.1361089243182925 km\n",
      "Title: Close to Madison Square Garden for games & events\n",
      "Document ID: BUn3hY8BHToGwgcUWuz5\n",
      "\n",
      "Distance to Empire State Building: 0.1361089243182925 km\n",
      "Title: Accessible-equipped, with 2 beds\n",
      "Document ID: S0n3hY8BHToGwgcUWuz5\n",
      "\n",
      "Distance to Empire State Building: 0.1370607600773514 km\n",
      "Title: A+ Location Superior Studio with 3 Beds\n",
      "Document ID: X0nxhY8BHToGwgcUFN0Q\n",
      "\n",
      "Distance to Empire State Building: 0.13903034558343705 km\n",
      "Title: Just What You Were Looking For! Pets Allowed\n",
      "Document ID: f0nUhY8BHToGwgcUvaLK\n",
      "\n",
      "Distance to Empire State Building: 0.1391429203202154 km\n",
      "Title: A+ Location Corner Deluxe Studio with 3 beds #7\n",
      "Document ID: eEnzhY8BHToGwgcUoeNe\n",
      "\n",
      "Distance to Empire State Building: 0.13945671370959836 km\n",
      "Title: Exclusive Private Studio 623 | Private Bathroom\n",
      "Document ID: iEn9hY8BHToGwgcUUfrA\n",
      "\n",
      "Distance to Empire State Building: 0.13953141538534714 km\n",
      "Title: Private Room 719 | Share bath\n",
      "Document ID: JEnehY8BHToGwgcUmLin\n",
      "\n",
      "Distance to Empire State Building: 0.13967442701583113 km\n",
      "Title: Rare Find! Near Museum of Modern Art\n",
      "Document ID: KEn3hY8BHToGwgcU2-18\n",
      "\n",
      "Distance to Empire State Building: 0.14136577000585135 km\n",
      "Title: Imperial Studios (Apt Selected at Check in #7\n",
      "Document ID: jkn2hY8BHToGwgcUnOqR\n",
      "\n",
      "Distance to Empire State Building: 0.14394037260994672 km\n",
      "Title: Private Room 541 | Shared bath\n",
      "Document ID: 6kn9hY8BHToGwgcU3_sk\n",
      "\n",
      "Distance to Empire State Building: 0.14475059845528132 km\n",
      "Title: Studio638 In the heart of Midtown Manhattan\n",
      "Document ID: 80nzhY8BHToGwgcU4ePP\n",
      "\n",
      "Distance to Empire State Building: 0.1447984852402037 km\n",
      "Title: *** HEART OF MANHATTAN (Herald Square) ***\n",
      "Document ID: bknyhY8BHToGwgcUX-D6\n",
      "\n",
      "Distance to Empire State Building: 0.14488109531419482 km\n",
      "Title: A+ Location Studio(APT Selected at Check-in) #4\n",
      "Document ID: N0nQhY8BHToGwgcU1JoO\n",
      "\n",
      "Distance to Empire State Building: 0.14520810510518647 km\n",
      "Title: 曼哈顿奢侈公寓次卧Manhattan luxury apartment second bedroom\n",
      "Document ID: pUoCho8BHToGwgcU8gfJ\n",
      "\n",
      "Distance to Empire State Building: 0.14632347379161145 km\n",
      "Title: Your Home Away From Home! 3 Convenient Units!\n",
      "Document ID: tUn_hY8BHToGwgcUIf4h\n",
      "\n",
      "Distance to Empire State Building: 0.14637379605706793 km\n",
      "Title: Next-Level Fun Only in NYC! Pets are Allowed!\n",
      "Document ID: Ikn0hY8BHToGwgcUY-UA\n",
      "\n",
      "Distance to Empire State Building: 0.14812006828137478 km\n",
      "Title: Cozy Scandinavian private room in Midtown\n",
      "Document ID: DEnGhY8BHToGwgcUk3-t\n",
      "\n",
      "Distance to Empire State Building: 0.14835485714012148 km\n",
      "Title: Сentered location*9th floor*studio for 5 people\n",
      "Document ID: sEn9hY8BHToGwgcUUfrA\n",
      "\n",
      "Distance to Empire State Building: 0.1487830005359137 km\n",
      "Title: Bunk(2full size) & 1 Full bed, next to KoreaTown10\n",
      "Document ID: gEndhY8BHToGwgcUy7Z6\n",
      "\n",
      "Distance to Empire State Building: 0.14921255320498628 km\n",
      "Title: Private Room 724 | Shared bath\n",
      "Document ID: I0nzhY8BHToGwgcUoeNe\n",
      "\n",
      "Distance to Empire State Building: 0.1492540503440509 km\n",
      "Title: A+ Location Comfort Lofty & Bright Queen Studio #5\n",
      "Document ID: 2UnFhY8BHToGwgcU1Hz2\n",
      "\n",
      "Distance to Empire State Building: 0.15110026929869902 km\n",
      "Title: Prime Location for Tourists! Near Times Square!\n",
      "Document ID: Kkn3hY8BHToGwgcU2-18\n",
      "\n",
      "Distance to Empire State Building: 0.1518058856616463 km\n",
      "Title: A+ Location City Studio with Two Full Beds\n",
      "Document ID: yEn9hY8BHToGwgcU3_sk\n",
      "\n",
      "Distance to Empire State Building: 0.15225274311294673 km\n",
      "Title: Near Times Square & Central Park! 3 Deluxe Units!\n",
      "Document ID: 4UnFhY8BHToGwgcUWnpw\n",
      "\n",
      "Distance to Empire State Building: 0.15418163408928168 km\n",
      "Title: Relax and Recharge! Four Units, Pets Allowed!\n",
      "Document ID: q0n_hY8BHToGwgcUIf4h\n",
      "\n",
      "Distance to Empire State Building: 0.1542895482106335 km\n",
      "Title: Manhattan Cozy Studio Near Empire State Building\n",
      "Document ID: VEnxhY8BHToGwgcUFN0P\n",
      "\n",
      "Distance to Empire State Building: 0.15436889515286376 km\n",
      "Title: Private Room 1616 | Shared bath\n",
      "Document ID: _En2hY8BHToGwgcUZum-\n",
      "\n",
      "Distance to Empire State Building: 0.1543975924221713 km\n",
      "Title: Your Own Studio Room Prime Midtown N833\n",
      "Document ID: 8knChY8BHToGwgcUyHLx\n",
      "\n",
      "Distance to Empire State Building: 0.1544440444562663 km\n",
      "Title: 4 people, center of the city\n",
      "Document ID: 1EnthY8BHToGwgcUCdS_\n",
      "\n",
      "Distance to Empire State Building: 0.15454404900499236 km\n",
      "Title: Exclusive Private Room A*13 | Shared Bathroom Only\n",
      "Document ID: pEn2hY8BHToGwgcUZum-\n",
      "\n",
      "Distance to Empire State Building: 0.154721101740187 km\n",
      "Title: Hotel 32 32, King Deluxe\n",
      "Document ID: GknRhY8BHToGwgcUY5vj\n",
      "\n",
      "Distance to Empire State Building: 0.15611194640292012 km\n",
      "Title: Exclusive Private Studio 1008 | Private Bathroom\n",
      "Document ID: 7EnIhY8BHToGwgcUwoSA\n",
      "\n",
      "Distance to Empire State Building: 0.15674564993112247 km\n",
      "Title: Great One Bedroom  Apartment Next  Empire State #9\n",
      "Document ID: TEnfhY8BHToGwgcUxLre\n",
      "\n",
      "Distance to Empire State Building: 0.15711486419513304 km\n",
      "Title: A+ Location Studio (APT Selected at Check-in) #5\n",
      "Document ID: qEnrhY8BHToGwgcUqNF5\n",
      "\n",
      "Distance to Empire State Building: 0.15751621275777955 km\n",
      "Title: NYC Vacay! Prime Location in Manhattan\n",
      "Document ID: _0n1hY8BHToGwgcUJ-an\n",
      "\n",
      "Distance to Empire State Building: 0.15891724131853355 km\n",
      "Title: Centered location Manhattan* 5th floor * 5 people\n",
      "Document ID: aUnBhY8BHToGwgcUkW7r\n",
      "\n",
      "Distance to Empire State Building: 0.15891724131853355 km\n",
      "Title: Centered location Manhattan* 5th floor * 5 people\n",
      "Document ID: LUnBhY8BHToGwgcU5nDB\n",
      "\n",
      "Distance to Empire State Building: 0.15892612790250735 km\n",
      "Title: 3beds next to Korea-town, Times Square2\n",
      "Document ID: FUnrhY8BHToGwgcUXdH0\n",
      "\n",
      "Distance to Empire State Building: 0.15900148684777546 km\n",
      "Title: Affordable in ENTiRE APT in heart of Manhattan.\n",
      "Document ID: S0nwhY8BHToGwgcUkNxN\n",
      "\n",
      "Distance to Empire State Building: 0.15930482830098514 km\n",
      "Title: A Place You'll Surely Enjoy! Pets are Allowed!\n",
      "Document ID: 0UnThY8BHToGwgcU-6D0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "json = answer.choices[0].text.split(\"```\")[1]\n",
    "# This now should contain just the json.\n",
    "json = JSON.loads(json)\n",
    "\n",
    "# first let's grab the location of the `near` field\n",
    "# it could be multiple locations, so we will search for all of them.\n",
    "near = client.search(\n",
    "    index=\"points-of-interest\",\n",
    "    size=100,\n",
    "    query={\"bool\": {\"must\": {\"terms\": {\"name\": [json[\"near\"].lower()]}}}},\n",
    ")\n",
    "\n",
    "# we store just all of the geo-locations of the near locations.\n",
    "near_location = []\n",
    "sort = []\n",
    "\n",
    "for hit in near[\"hits\"][\"hits\"]:\n",
    "    near_location.append(hit[\"_source\"][\"location\"])\n",
    "    sort.append(\n",
    "        {\n",
    "            \"_geo_distance\": {\n",
    "                \"location\": {\n",
    "                    \"lat\": hit[\"_source\"][\"location\"][\"lat\"],\n",
    "                    \"lon\": hit[\"_source\"][\"location\"][\"lon\"],\n",
    "                },\n",
    "                \"unit\": \"km\",\n",
    "                \"distance_type\": \"plane\",\n",
    "                \"order\": \"asc\",\n",
    "            }\n",
    "        }\n",
    "    )\n",
    "\n",
    "query = {\n",
    "    \"geo_distance\": {\n",
    "        \"distance\": str(json[\"distance_in_km\"]) + \"km\",\n",
    "        \"location\": {\"lat\": near_location[0][\"lat\"], \"lon\": near_location[0][\"lon\"]},\n",
    "    }\n",
    "}\n",
    "# Now let's get all the AirBnBs `what` near the `near` location.\n",
    "# We always use the first location as our primary reference.\n",
    "airbnbs = client.search(index=\"airbnb-*\", size=100, query=query, sort=sort)\n",
    "\n",
    "for hit in airbnbs[\"hits\"][\"hits\"]:\n",
    "    print(f\"Distance to {json['near']}: {hit['sort'][0]} km\")\n",
    "    print(f\"Title: {hit['_source']['name']}\\nDocument ID: {hit['_id']}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With that we now have combined geo spatial search with LLMs.\n",
    "\n",
    "Some idea for further exploration:\n",
    "* Let any LLM generate an itinerary with sights.\n",
    "..."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
